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USER  EVALUATION  OF  THE  UTILITY  OF  TWELVE  DECISION  AIDS 


Introduction 

With  the  advance  of  computer  technology,  more  decision  aids 
are  being  used  in  the  military  environment.  Various  decision 
aids  have  been  developed  specifically  to  assist  in  communication 
and  decision  making  processes  for  the  military.  Twelve  such  aids 
were  demonstrated  or  utilized  during  a  training  exercise.  Prairie 
Warrior  94,  at  Ft.  Leavenworth,  Kansas.  This  report  documents 
the  assessment  of  utility  of  these  decision  aid  systems  by 
student  users  and  government  and  contractor  evaluators.  The 
assessment  employed  a  generic  user  questionnaire  developed  to 
evaluate  the  utility  of  decision  aids  at  various  stages  of 
development . 


Description  of  Generic  Utility  Questionnaire 

The  questionnaire  that  was  employed  to  assess  utility  was 
developed  to  be  easily  tailored  to  different  aids  and  different 
development  stages .  The  generic  questionnaire  was  developed  in 
four  steps  (Adelman,  Gualtieri,  &  Riedel,  1993) .  First,  the 
developers  reviewed  the  literature  to  identify  the  different 
attributes  of  utility  defined  in  the  literature.  Second,  a 
Multi-Attribute  Utility  Assessment  (MAUA)  hierarchy  (Fi^re  1) 
was  devised  for  combining  the  individual  usability  attributes 
into  broader  utility  concepts.  Third,  two  or  more  questions  were 
developed  to  measure  each  bottom-level  attribute  of  the 
hierarchy.  Last,  the  questionnaire  was  pilot -tested  to  determine 
its  psychometric  characteristics.  The  development  and  validation 
of  the  instrument  is  described  in  A.delman,  Gualtieri,  Riedel  and 
Trent  (1996) . 

The  MAUA  hierarchy  depicts  the  nodes  that  make-up  the 
utility  of  the  decision  aid.  The  top  level  nodes  are  system  fit, 
system  usability  and  effect  on  task  performance.  Each  of  these 
dimensions  is  then  further  divided  into  specific  attributes 
(Adelman,  et  al . ,  1995). 

System  fit  is  the  fit  of  the  system  with  the  organization 
and  user.  Attributes  of  system  fit  are  how  well  the  aid  fits 
with  the  flow  of  information,  with  the  user's  needs,  with  problem 
solving,  with  doctrine  and  with  others'  work.  System  usability 
is  subdivided  into  ease  of  training,  general  ease  of  use, 
workload,  quality  of  user's  mental  model,  human  factors, 
flexibility  and  allocation  of  functions  between  the  person  and 
machine.  Product  and  process  quality  are  attributes  of  the 
effect  on  task  performance  node  (Adelman,  et  al.,  1995)  . 
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Figure  1.  A  MAUA  Evaluation  Hierarchy  for  Assessing 
Users’  Opinions  about  System  Utility. 


Description  of  Decision  Aid  Systems 

The  functions  of  the  decision  aids  varied  from  visualization 
of  the  battlefield  to  exchange  of  information  to  weather 
analysis.  The  aids  that  supported  visualization  of  the 
battlefield  are;  Commander's  Visualization  Research  Tool, 

Flying  Carpet,  Magic,  Mission  Planning  Rehearsal  System  ,  and 
Multi-Spectral  Capability.  The  Battle  Command  Decision  Support 
System  assisted  with  unit  analysis,  the  weather  was  analyzed  by 
the  Integrated  Weather  Effects  Decision  Aid,  Mapinfo  Desktop 
Mapping  Decision  Aid  provided  map  overlays,  avenues  of  approach 
were  determined  by  the  METT-T  aid,  and  logistics  were  assessed  by 
the  KELPS  aid.  Information  exchange  was  aided  by  E-Note  and 
Video  Teleconferencing  support  systems. 

Battle  Command  Decision  Support  System  (BCDSS) .  This  is  a 
visual  aid  that  displays  unit  status .  Its  intent  is  to  provide 
the  cotrmander  with  a  quick  visual  reference  on  unit  status.  The 
commander  and  staff  are  also  provided  with  an  interface  for 
looking  at  facts  and  figures  that  support  the  visual  displays. 

The  system  supports  mission  analysis,  course  of  action  (COA) 
development,  COA  analysis,  decision  and  execution. 

Commander's  Visualization  Research  Tool  (CoVRT) .  This  aid 
was  developed  to  support  the  commander  and  intelligence  analyst. 
It  provides  a  real  time  display  of  messages,  graphics  and 
imagery.  The  system  is  compatible  with  the  Maneuver  Control 
System  (MCS)  and  supports  a  shared  visualization  of  the 
battlefield  between  Brigade  Commander  and  staff.  It  provides  a 
real  time  display  of  messages,  graphics  and  imagery,  and  assists 
in  the  development  and  dissemination  of  intelligence 
requirements.  Incorporated  within  this  system  is  a  wide  variety 
of  graphics  tools  to  facilitate  overlay  preparation. 

Electronic  Mail  (E-NOTE) .  E-Note  is  an  electronic  mail 
system  that  is  designed  to  facilitate  the  exchange  of  information 
and  staff  coordination.  It  allows  both  point-to-point  and  point- 
to-many  distribution  over  local  area  and  wide  area  networks  (LANs 
and  WANs) .  Briefing  slides  and  documents  can  be  attached  to  text 
messages.  It  supports  COA  development  and  mission  execution 
tasks . 

Flying  Carpet .  This  is  a  terrain  visualization  tool  for  the 
commander  and  staff  to  assist  in  mission  analysis.  Intelligence 
Preparation  of  the  Battlefield  (IPB) ,  wargaming  and  monitoring 
own  forces.  It  provides  the  user  with  3-D  fly  through 
visualization  of  the  battlefield,  3-D  view  of  the  terrain  from  a 
specified  vehicle,  and  realistic  battle  sounds.  It  has  the 
capability  to  play  out  a  battle  given  a  mission  and  unit 
taskings . 


3 


Integrated  Weather  Effects  Decision  Aid  (IWEDA) .  This 
system  provides  commanders,  staff,  and  staff  weather  officers 
with  weather  effects  information,  including  the  impacts  on 
missions,  systems,  subsystems  and  components.  It  also  provides 
detailed  explanations  on  why  conditions  are  marginal  or 
unfavorable,  and  enables  the  user  to  examine  alternative  mission 
set  ups  and  weather  conditions.  Another  feature  is  the  provision 
of  current  and  forecast  weather  for  geographic  area  of  interest. 

Knowledge  Based  Logistic  Planning  System  (KELPS)  .  This  _ 
logistics  planning  tool  was  developed  for  use  by  the  logistics 
officer  (G4)  and  staff.  It  has  the  ability  to  compare 
requirements  with  log  capabilities,  has  mapping  capabilities,  and 
calculates  consumption  of  weapon  systems  over  time.  The  system 
estimates  Class  III  (fuels  and  lubricants)  and  V  (ammunition) 
requirements  given  specified  planning  factors,  and  displays  log 
inventory  by  echelon. 

magic.  The  purpose  of  MAGIC  is  to  enable  the  commander  to 
see  the  battlefield  terrain  and  share  a  common  view  of  the 
battlefield  with  the  staff.  The  features  include  its  ability  to 
provide  a  real  time,  high  resolution  3-D  fly-drive  through  of 
terrain  and  display  of  moving  vehicles.  Additionally,,  the 
system  has  access  to  terrestrial  and  satellite  communication 
networks,  and  aerial  photography.  It  allows  the  user  to  control 
the  view  point  and  route,  and  supports  mission  planning, 
rehearsal,  IPB,  and  execution. 

Mapinfo  Desktop  Mapping  Decision  Aid  (MAPINFO) .  The  system 
is  a  desktop  mapping  tool  and  decision  aid  which  supports  the 
display  of  maps  and  overlay  graphics.  It  is  a  geographic 
information  system  (GIS)  which  can  display  Battle  Command  data  by 
location.  User  data  is  displayed  in  the  correct  location  with 
appropriate  colors  and  symbols.  The  data  may  be  viewed  in  text, 
map  or  graphic  format.  Data  behind  any  object  on  a  map  may  be 
queried  and  the  distance  between  points  calculated. 

Mission.  Enemy,  Trooos.  Terrain  and  Time  Available  (METT-T) . 
This  system  assists  staff  officers  with  the  analysis  of  avenues 
of  approach  and  determining  the  tactical  effects  of  terrain  and 
weather  on  combat  operations.  Additionally,  it  aids  these 
officers  in  determining  and  developing  task  organizations _ and 
combat  capabilities  for  both  current  and  projected  situations. _ 

It  also  assists  with  analyzing  enemy  capabilities  and  determining 
projected  threat  intent. 

Mission  Planning  Rehearsal  System  (MPRS) .  Through  terrain 
"computer  visualization"  of  the  battlefield,  MPRS  supports  battle 
command.  It  also  supports  Terrain  Analysis,  IPB  and  other 
Intelligence  Analysis  functions.  MPRS  also  sustains  the  tasks  of 
mission  analysis,  COA  development,  COA  analysis  and  comparison, 
decisions  and  execution. 
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Multi-Spectral  Capability  (MSI) .  This  decision  aid  assists 
planners  to  perform  terrain  and  intelligence  analyses  (including 
Intelligence  Preparation  of  the  Battlefield  (IPB))  in  support  of 
mission  analysis,  COA  development,  and  COA  analysis  and 
comparison.  Two  other  features  are  an  automated  mapping  aid  to 
facilitate  staff  operations  and  the  ability  to  produce  maps  using 
satellite  imagery  where  maps  or  DMA  data  are  not  immediately 
available . 

Space  Enhanced  Command  and  Control  Capabilities  (SPECC) . 

This  device  supports  automated  worldwide  unit  location  functions. 
Its  features  include:  World  wide  automated  unit  locations,  timing 
synchronization  (Worldwide  Atomic  Clock) ,  multimap  display, 
worldwide  asset  tracking  (GPS  &  Tracker) ,  worldwide 
interconnectivity  (Satellite  Communications) ,  Intelligence 
broadcast  reception  (National  Asset  Downlink) ,  Situational 
awareness  (red  and  blue  force  status) ,  Operation  Order  (OPORD) 
generat'ion  and  transmission,  synchronization  matrices,  Unit-Log- 
Personnel  and  Status  reports,  joint  forces  Command  and  Control 
(C2)  applicability,  and  other  C2  activities.  Its  primary  purpose 
is  to  aid  staff  members  in  performing  mission  analysis,  IPB,  and 
other  intelligence  functions,  COA  analysis  and  comparison, and 
execution  tasks . 

Video  Teleconferencing  (VTC) .  An  aid  designed  to  facilitate 
information  exchange  and  staff  coordination.  VTC  is  a 
collaboration  tool  that  allows  two  or  more  people,  remotely 
located,  to  talk  face-to-face  in  real-time.  Staff  members  are 
aided  in  performing:  Mission  analysis,  Intelligence  Preparation 
of  the  Battlefield  (IPB)  and  other  intelligence  functions,  COA 
analysis  and  comparison,  and  execution  tasks. 
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Method 


Subjects 

Twenty  four  Command  and  General  Staff  College  students 
enrolled  in  a  course  testing  command  and  control  innovations  and 
five  government  or  contract  employees  serving  as  data  collectors 
were  asked  to  complete  questionnaires  addressing  the  utility  of 
twelve  decision  support  systems.  This  was  accomplished  after  the 
students  had  participated  in  a  week  long  exercise  in  which  they 
used  the  aids.  All  data  collectors  and  fourteen  students 
completed  at  least  one  of  the  questionnaires. 

Questionnaire 

On  the  first  page  of  each  questionnaire  was  a  description  of 
the  functions  that  were  provided  by  the  aid,  directions  for 
complet'ing  the  form,  a  request  for  demographic  data  and  questions 
asking  for  the  number  of  hours  of  experience  with  the  aid  and  the 
nature  of  the  interaction  with  the  aid  (direct  an  operator, 
operate,  observe) . 

For  the  evaluation  described  in  this  report,  staff  from  the 
Battle  Command  Training  Program  (BCTP)  at  Fort  Leavenworth, 
Kansas,  selected  the  attributes  from  the  Standard  Utility 
hierarchy  that  fit  each  attribute  they  wanted  evaluated. 
Appropriate  questions  were  then  selected  and  a  questionnaire  was 
specifically  tailored  for  each  decision  aid.  For  this  section  of 
the  questionnaire,  a  7 -point  Likert  scale  was  used  to  indicate 
degree  of  agreement  with  the  statement.  Due  to  a  stipulation  by 
BCTP,  the  students'  questionnaires  were  composed  of  fewer 
questions  than  the  ones  for  the  data  collectors.  Questionnaires 
for  the  different  aids  varied  in  length  because  of  differences  in 
type  and  stage  of  development  of  the  aid.  Example  questionnaires 
for  the  two  groups  are  in  Appendices  B  and  C. 

A  second  section,  which  differed  in  length  and  complexity 
for  students  and  data  collectors,  required  an  assessment  of  the 
capabilities,  advantages  and  disadvantages  of  the  aid.  One  part 
of  this  section  asked  the  respondents  (students  and  data 
collectors)  to  signify,  using  a  5-point  Likert  scale,  the  extent 
that  the  aid  would  improve  battle  planning  and  execution,  mission 
analysis,  intelligence  preparation  of  the  battlefield _  (IPB) , 
course  of  action  development,  course  of  action  analysis  and 
comparison,  the  commander's  decision  and  execution. 

Questionnaires  for  the  data  collectors  also  included  a  rating  of 
how  much  the  capabilities  of  each  aid  would  increase  situational 
awareness,  facilitate  a  shared  understanding  of  the  battlefield, 
assist  in  the  synchronization  of  tactical  operations  and 
-  facilitate  plan  rehearsal .  The  data  collectors  were  asked  to 
rate  (1)  how  much  the  tested  aid  capabilities  could  potentially 
improve  performance  and  (2)  how  much  the  aid  actually  improved 
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performance.  Additional  questions  requested  which  capabilities 
should  be  eliminated  or  added,  advantages  and  disadvantages  of 
the  aid,  and  flexibility  of  the  aid  to  perform  different  tasks. 
All  questionnaires  were  similar  in  design  and  focused  on  the 
utility  of  the  systems  in  the  military  environment .  One 
question,  addressing  the  extent  to  which  the  aid  could  replace 
paper  map  functions  in  the  tasks  of  planning,  analysis, 
execution,  and  command  and  control,  was  included  for  MPRS  and 
MSI.  Finally,  students  and  data  collectors  were  encouraged  to 
provide  comments  on  any  aspect  of  the  aid. 

All  aids  were  not  rated  by  all  of  the  students  or  data 
collectors.  The  reasons  for  this  are:  1)  the  aid  was  not 
functional  (KBPLS) ,  2)  the  aid  did  not  fit  into  what  the  student 
was  doing,  3)  the  rater  only  observed  the  use  of  the  aid  and  did 
not  feel  comfortable  evaluating  it .  In  some  instances  only  one 
student  or  data  collector  responded  to  the  questionnaire  and  all 
ratings'* should  be  viewed  with  this  in  mind.  Of  the  students,  one 
student  rated  CoVRT,  Flying  Carpet,  Magic  and  METT-T;  and  of  the 
data  collectors  one  data  collector  rated  IWEDA,  Magic,  METT-T  and 
MSI. 
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RESULTS 


The  utility  of  each  of  the  aids  was  evaluated  by  students 
and  data  collectors.  The  tables  that  follow  display  the  means, 
standard  deviations  and  range  for  each  attribute  in  the  utility 
hierarchy.  A  synopsis  of  the  comments  related  to  each  aid  are 
included  before  each  table.  The  overall  utility  in  the  tables  is 
based  on  questions  of  overall  utility  and  is  not  based  on  an 
integration  of  the  subattribute  scores. 

The  ratings  are  based  on  responses  to  questions  employing  a 
7  point  Likert  scale,  where  1  means  the  system  has  a  very 
negative  effect,  4  means  the  system  is  neither  negative  nor 
positive  and  7  that  the  system  is  rated  very  positively.  The 
mean,  over  all  respondents,  for  overall  utility  was  greater  than 
one  for  all  aids,  suggesting  all  aids  were  seen  as  having  some 
utility. 


Battle  Command  Decision  Support  System  (BCDSS) 

The  purpose  of  this  aid  is  to  provide  a  visual 
representation  of  the  unit  status  for  the  Commanders  and  Staff 
Officers.  Two  students  observed  the  aid  as  it  was  being  used, 
and  the  others  operated  the  aid.  An  examination  of  the  following 
table  (See  Table* 1)  for  student  assessment  of  utility  indicates 
that  the  mean  for  overall  utility  was  4.95  with  little 
variability  among  the  students'  ratings.  All  data  collectors 
observed  the  operation  of  the  decision  aid.  The  ratings  (See 
Table  2)  of  the  data  collectors  for  this  overall  utility  resulted 
in  a  mean  of  5.30,  again  with  little  variability.  This  indicates 
that,  for  the  most  part,  the  two  groups  believed  that  the  aid  had 
some  utility. 

As  reflected  in  Table  1  and  2,  the  students'  and  data 
collectors'  evaluation  of  the  quality  of  the  results  demonstrated 
confidence  in  the  results  with  little  variability  of  their 
opinions.  The  ease  of  use,  human  factors  and  flexibility  of  the 
aid  received  the  highest  ratings  by  the  students.  However,  the 
data  collectors  rated  human  factors  lowest  of  the  three 
dimensions . 

The  most  variability  and  consistently  lowest  ratings  of  the 
students  were  found  with  those  questions  addressing  the  system 
fit  with  the  organization  and  user.  System  fit  refers  to  the 
degree  to  which  the  characteristics  of  the  system  correspond  with 
the  needs  of  the  users  and  their  organization.  It  is  considered 
to  be  a  measure  of  how  much  the  system  will  be  utilized.  This 
variability  was  not  found  for  the  data  collectors  who,  as  a 
group,  rated  these  characteristics  higher  than  the  students' 
ratings . 
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Differences  in  the  results  of  the  ratings  for  the  two  groups 
can  be  accounted  for  by  the  difference  in  the  number  of  students 
(7-10)  and  data  collectors  (3)  responding  to  the  questionnaire. 
There  was  consistently  less  variability  among  the  ratings  for  the 
data  collectors  than  the  students.  Another  reason  is  that  the 
data  collectors  could  have  initially  been  more  familiar  with  the 
aid,  or  entered  the  evaluation  process  with  biases  toward  the 
aid.  Additionally,  the  data  collectors  were  not  operating  the 
aid  as  a  device  for  gaining  information  for  the  exercise  but 
rather  obseirving  use  of  its  capabilities.  Six  respondents 
commented  that  automated  data  update  was  needed  if  BCDSS  was  to 
be  useful. 

Although  the  ratings  were  above  the  midpoint  for  all  of  the 
characteristics  assessed,  students  and  data  collectors  had 
several  negative  comments  concerning  the  aid.  Many  raters  from 
each  group  remarked  that  the  aid  was  slow,  cumbersome  at  times, 
increased  user  workload,  and  was  too  difficult  to  maintain. 


Commander's  Visualization  Research  Tool  (CoVRT) 

This  aid  supports  a  shared  visualization  of  the  battlefield 
between  the  brigade  commander  and  staff  and  provides  a  real  time 
display  of  messages,  graphics,  and  imagery.  Five  students 
returned  questionnaires,  but  only  one  student  evaluated  the  aid 
(See  Table  3).  Another  student  said  he  only  it.  Three  other 
students  who  returned  questionnaires  indicated  CoVRT  was 
unavailable  or  they  had  not  used  or  observed  it.  As  noted  in  the 
table  below,  the  one  student  who  evaluated  CoVRT  rated  all 
attributes  addressed  on  the  questionnaire  above  the  midpoint 
marker  without  commenting  on  any  areas . 

The  mean  rating  for  overall  utility  for  three  data 
collectors  (See  Table  4)  was  3.76.  This  is  in  the  marginal  range 
of  performance  range  of  the  utility  scale.  Data  collectors  rated 
CoVRT  as  difficult  to  use  giving  the  system  unfavorable  ratings 
for  ease  of  use.  Human  factors  and  flexibility.  The  highest  mean 
ratings  were  for  the  quality  of  problem  solving  and  system 
quality,  with  both  scoring  just  above  4. 

The  difference  in  the  number  of  data  collectors  and  students 
who  evaluated  this  aid  make  it  difficult  to  compare  and  contrast 
the  ratings  of  each  group.  The  student  rater  did  not  express  any 
displeasure  with  the  aid.  One  data  collector  commented  that  in 
general  CoVRT  would  require  a  great  deal  of  training.  In 
addition,  use  of  the  raw  imagery  data  the  system  provided  would 
require  the  assistance  of  an  imagery  analyst. 

■  E-NOTE 

E-Note  is  an  electronic  mail  system  that  is  designed  to 
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Student  Utility  Scores  for  E-NOTE 


facilitate  the  exchange  of  information  and  staff  coordination.  It 
supports  COA  development  and  mission  execution  tasks .  E-Note  was 
one  of  the  most  used  of  the  fourteen  decision  aids  examined  in 
Prairie  Warrior;  twelve  students  reported  they  had  used  E-Note  in 
the  exercise.  It  generally  scored  well  on  the  questionnaire 
although  respondents  had  many  suggestions  for  improvement.  An 
examination  of  their  ratings  (See  Table  5)  indicated  that  the 
highest  mean  ratings  were  for  quality  of  the  results,  system 
quality  and  quality  of  problem  solving.  The  mean  ratings  of 
these  were  all  above  5,  indicating  users'  perceived  E-Note  as 
useful.  However,  they  rated  ease  of  use  and  system  fit  with  the 
user  and  organizations  as  marginal. 

Data  collectors'  ratings  were  higher  than  the  students' 
ratings  for  all  the  attributes.  They  also  rated  E-Note  as 
marginal  on  ease  of  use . 

Students  and  data  collectors  had  many  comments  about  the 
disadvantages  of  the  system  and  suggestions  for  improvement.  Two 
students  noted  that  this  aid  increases  the  workload  of  the  staff 
because "the  time  it  takes  to  type,  retrieve,  move  notes  around, 
etc,  is  all  wasted  time",  and  lacks  "a  protocol  that  will 
establish  files  and  help  route  info".  Other  comments  addressed 
an  interruption  to  work  and  other  functions  because  the  notes 
"pop  up  into  the  middle  of  the  work".  Difficulty  was  also 
indicated  in  the  facilitation  of  the  flow  of  information  during 
planning  and  execution.  One  student  felt  that  this  system  would 
assist  in  this  area  if  "some  one  is  dedicated  to  monitoring  the 
system  and  bringing  important  messages  to  the  attention  of  the 
commander,  etc . " 

The  primary  focus  of  the  data  collectors'  comments  indicated 
a  lack  of  functionality  of  the  system  because  of  a  need  for 
"better  logical  icons",  a  militarized  version,  and  a  "fully 
functional  e-mail  system" .  There  were  also  comments  that  the 
system  was  rudimentary  and  needed  "a  more  sophisticated  and  user 
friendly  capability  for  it  to  be  a  true  asset  and  meet  user 
acceptability" . 

Elvina  Carpet 

This  is  a  terrain  visualization  tool  which  provides  a  3-D 
visualization  of  the  battlefield,  terrestrial  and  satellite 
communication  networks,  and  aerial  photography.  This  decision 
aid  was  not  available  to  the  group  for  use,  but  was  demonstrated. 
One  student  and  two  data  collectors  responded  to  the 
questionnaire,  and  the  results  are  shown  in  Tables  7  and  8. 
Overall,  data  collectors  rated  Flying  Carpet  as  marginal  in 
overall  quality  with  ease  of  use  as  marginal  or  deficient.  The 
student  who  rated  this  system  thought  it  had  marginal  to  good 
overall  quality  but  deficient  in  its  fit  with  organization  needs. 


13 


Student  Utility  Scores  for  FLYING  CARPET 


Comments  made  by  the  data  collectors  indicated  that  the  aid 
needed  to  be  more  user  friendly,  it  is  physically  too  large  to 
fit  into  the  operations  of  the  staff,  and  it  needs  some 
enhancements  for  the  user  to  have  confidence  in  its  approach  to 
planning.  Great  visualization  of  the  commander's  battle  space 
was  viewed  as  an  advantage.  The  student  stated  that  the  aid  was 
more  like  a  toy  than  a  tool . 

IWEDA 


This  system  provides  commanders,  staff,  and  staff  weather 
officers  with  weather  effects  information,  and  detailed 
explanations  on  why  conditions  are  unfavorable  for  the  missions 
and  systems .  The  four  students  who  completed  this  questionnaire 
gave  the  highest  ratings  (See  Table  9)  to  fit  with  the 
organization,  quality  of  results,  system  quality,  and  the  overall 
utility  rating.  Although,  students  rated  system  fit  and  fit  with 
organization  above  5  points,  their  rating  of  fit  with  user  (4.75) 
was  slightly  lower.  Among  these  ratings,  fit  with  organization 
had  the  least  variability,  .76.  The  lowest  rating  (3.5)  was 
given  to  flexibility  of  the  system,  which  also  had  the  most 
variability  (2.12).  One  data  collector  rated  (See  Table  10)  this 
aid,  rating  most  of  the  dimensions  at  around  5.  The  students 
commented  that  IWEDA  did  not  tell  much  more  than  could  be  gotten 
from  the  staff  weather  officer  (SWO) ,  it  was  not  worth  the  space 
that  would  be  taken  up  in  the  TOC,  and  the  system  needs  too  much 
user  input.  Additionally,  "IWEDA  is  a  system  that  tells  a 
commander  the  effects  of  Wx  (weather)  on  his  personnel  and  weapon 
systems".  One  student  said  this  was  a  no-brainer  for  a 
"competent  commander."  Although  receiving  fairly  consistent 
midpoint  ratings,  these  remarks  indicate  that  this  aid  was  not 
considered  to  have  much  utility  for  the  staff.  Suggested 
improvements  included  automated  weather  data  entry,  effects  on 
enemy  systems,  and  links  to  other  army  systems. 

Some  comments  reflected  positive  attributes  of  the  aid.  The 
shift  between  the  big  picture  and  specifics  was  good,  and  it  has 
good  forecast  options.  One  student  wrote  that  the  device  needs 
to  address  three  things:  1)  current  weather,  2)  the  forecast,  3) 
effects  on  enemy  weapon  systems.  However,  another  stated  that 
the  aid  was  not  user  friendly,  and  a  large  screen  was  needed. 

KELPS 


Was  not  operational. 
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MAGIC 


MAGIC  allows  the  commander  to  view  the  battlefield  terrain 
and  to  share  a  common  view  of  the  battlefield  with  his  staff. 
One  student  and  no  data  collectors  responded  to  the 
questionnaire.  The  student's  ratings  are  listed  in  Table  11. 
Overall  the  student  rated  this  system  as  having  a  great  deal  of 
utility.  He  did  not  rate  the  ease  of  use  of  the  system. 
Comments  that  were  made  were  generally  positive.  Comments 
include  that  it  is  "potentially  good  for  terrain",  and  provides 
useful  information  to  the  commander,  but  that  the  lag  times  for 
response  to  user's  commands  were  too  long  to  create  the  views. 
Because  only  one  student  responded  to  the  questionnaire,  the 
results  do  not  provide  a  useful  appraisal  of  the  utility  of  the 
aid. 

MAP INFO 


This  aid  supports  the  display  of  maps  and  overlay  graphics. 
Twelve  students  and  all  of  the  data  collectors  responded  to  this 
questionnaire .  Map  Info  was  the  only  system  that  was  used  by  all 
the  students.  It  also  received  their  lowest  ratings  of  all  the 
aids  on  overall  utility,  system  quality  and  system  fit  with  the 
user.  Conversely  the  ratings  (See  Table  13)  of  the  data 
collectors  were  above  the  midpoint  for  all  dimensions.  As  with 
the  students,  usability  was  rated  as  marginal,  but  MAPINFO  was 
rated  as  having  utility  in  terms  of  quality  of  the  system  and 
system  fit. 

Comments  of  the  students  ranged  from  "lots  of  potential"  but 
needs  "fixes"  concerning  planning  and  execution  performance  to 
indications  that  the  speed  was  slow  and  cumbersome . 

Additionally,  four  of  the  students  reported  that  the  aid  would 
increase  the  workload.  One  student  commented  that  he  was  "never 
less  than  frustrated  with  Mapinfo"  and  another  that  the  aid  needs 
"lots  of  fixes"  and  a  third  "to  get  rid  of  Mapinfo". 

The  data  collectors  comments  included  that  the  aid  had  poor 
functionality,  it  was  dependent  on  the  source  of  data,  and  the 
user  needed  the  FM  (field  manual)  on  operational  graphics. 
Additionally,  it  was  not  viewed  as  being  easy  to  use  and  required 
too  many  keystrokes . 

Suggestions  for  improvement  included  maps  with  additional 
graphics,  standard  map  scales  and  special  terrain  data  like 
obstacles;  a  better  user  interface,  and  links  to  systems  like 
ASAS  for  enemy  templates  and  MSI  for  satellite  data. 
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METT-T  (Mission,  Enemy,  Troops.  Terrain-Time  Available) 


This  system  assists  staff  officers  with  the  analysis  of 
avenues  of  approach  and  in  determining  the  tactical  effects  of 
terrain  and  weather  on  combat  operations .  One  student  and  one 
data  collector  responded  to  the  questionnaire.  The  results  are 
presented  in  Tables  14  and  15 ,  The  one  student  who  did  rate  the 
aid  had  observed  a  demonstration  of  its  use.  He  commented  that 
it  did  not  appear  to  be  user  friendly,  and  that  he  was  not  able 
to  make  comments  on  other  aspects  of  this  device  unless  he  could 
use  it . 

The  data  collector  rated  the  system  as  having  very  good 
interface.  He  thought  the  concept  for  METT-T  was  good  but  that 
the  system  had  marginal  utility  at  this  time . 

Mission  Planning  Rehearsal  System  (MPRS) 

MPRS  is  a  decision  aid  that  supports  Battle  Command  through 
terrain  "computer  visualization"  of  the  battlefield.  The  six 
students  who  rated  this  device  had  observed  its  use.  Their 
ratings  (See  Table  16)  were  consistently  above  the  midpoint  for 
all  dimensions  with  the  quality  of  problem  solving  rated  as  very 
good . 


Three  of  the  five  data  collectors  completed  the 
questionnaire  (See  Table  17) .  The  highest  rating  was  also  given 
to  the  quality  of  problem  solving  of  the  device  (5.42)  .  Both 
students  and  data  collectors  felt  the  system  was  too  slow.  Their 
ratings  on  all  dimensions  of  the  aid  were  consistently  lower  than 
those  given  by  the  students.  Several  commented  that  better 
terrain  resolution  was  needed,  that  the  system  should  be  made 
more  user  friendly,  and  that  the  complex  Unix  system  would 
require  a  great  deal  of  user  training. 

Multi-Soectral  Capability  (MSI) 

This  decision  aid  provides  terrain  visualization  and  assists 
planners  to  perform  terrain  and  other  intelligence  analyses. 

Eight  students  and  one  data  collector  responded  to  this 
questionnaire.  The  students  rated  (See  Table  18)  the  quality  of 
problem  solving  and  fit  with  the  organization  as  good  and  the 
usability  factors  as  marginal. 

One  student  felt  the  aid  had  potential  because  it  offered 
"other  perspectives  that  are  useful",  and  another  commented  that 
it  is  useful  because  it  brings  mapping  capabilities  into  unmapped 
parts  of  the  world.  Other  respondents  felt  the  aid  needed  better 
resolution,  operational  graphics,  a  ATM  grid  overlay,  a  better 
interface  and  to  be  linked  to  other  systems. 
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Space  Enhanced  Command  and  Control  Capabilities  (SPECC) 


This  device  supports  automated  worldwide  unit  location 
functions.  Twelve  students  and  three  data  collectors  rated  this 
aid;  of  these,  four  students  had  used  the  aid.  The  students 
rated  (See  Table  20)  the  quality  of  results  and  system  fit  as 
good,  and  usability  as  marginal. 

The  three  data  collectors  rated  system  quality,  system  fit 
flexibility,  and  usability  as  good.  Quality  of  problem  solving, 
a  subattribute  of  system  quality,  was  rated  very  good  by  the  data 
collectors.  Of  interest  is  the  difference  in  ratings  for 
flexibility  between  the  students  and  data  collectors.  The 
students  viewed  this  aid  has  having  less  flexibility  than  did  the 
data  collectors. 

As  reflected  in  their  comments,  the  students  felt  that  this 
aid  was'^slow  to  build  operational  graphics,  not  user  friendly  and 
needed  better  map  resolution.  One  data  collector  expressed 
concern,  in  several  instances,  about  the  amount  of  user  training 
that  the  aid  would  require.  This  person  also  felt  that  the  aid 
was  not  user  friendly,  although  it  did  have  a  "good  graphic  map 
picture" .  Other  suggestions  for  improvement  include 
interconnectivity  with  satellite  communications  and  ASAS,  and 
that  capabilities  such  as  unit  status  display,  task  organization, 
and  sync  metrics  be  added. 

Video  Teleconferencing  (VTC) 

VTC  is  a  collaboration  tool  that  allows  two  or  more  people, 
remotely  located,  to  talk  face-to-face  in  real-time.  Ten 
students  and  three  data  collectors  rated  this  aid.  VTC  received 
the  highest  ratings  of  any  of  the  aids  across  all  of  the  rating 
dimensions.  The  students  and  data  collectors  viewed  (See  Table 
22  and  23)  the  quality  of  problem  solving,  system  quality  and  fit 
with  the  organization  as  good  to  very  good.  Usability  dimensions 
were  rated  somewhat  lower  but  still  in  the  good  range. 

The  students'  comments  indicated  that  this  aid  would  not 
replace  face-to-face  communication,  will  increase  work  load  for 
some,  and  needs  to  be  able  to  conference  VTC  with  about  8-12 
participants.  Students  also  thought  a  direct  feed  of  computer 
maps  and  graphics  would  aid  the  resolution  of  these  pictures.  An 
advantage  was  seen  in  that  the  aid  required  little  training. 
Student  recommendations  were  to  incorporate  an  integrated  fax  to 
out  stations  and  to  have  miniature  systems  between  Army  posts. 

The  data  collectors  also  found  that  users  of  the  aid  needed 
little  training,  that  the  capabilities  were  diverse  and  could 
augment  face-to-face  communication.  Disadvantages  were  noted  in 
the  resolution  of  the  picture,  it  needed  to  be  more  user 
friendly. 
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Discussion 


Overall,  the  highest  ratings  were  given  to  the  aids  that 
assisted  in  communications.  Eleven  students  rated  VTC  highest 
on  all  but  the  dimensions  of  flexibility  and  quality  of  results. 
VTC  was  viewed  more  positively  across  all  the  rated  dimensions 
than  any  other  aid.  One  reason  for  this  is  that  video 
teleconferencing  is  familiar  to  the  officers  because  it  is 
already  in  use.  Added  to  this  is  video  teleconferencing  is 
considered  to  be  user  friendly.  There  was  little  difference 
between  the  data  collectors  and  student  ratings  for  this  aid. 

The  results  of  this  study  show  there  is  a  need  for  the 
capabilities  represented  in  the  evaluated  aids.  Video 
teleconferencing,  map  displays,  creating  and  showing  operational 
graphics,  unit  status  display  and  update,  automated  terrain 
analysis--these  and  more  have  the  potential  to  improve  tactical 
planning  and  operations. 

However,  many  improvements  are  needed  before  the  aids  will 
realize  that  potential.  One  of  the  most  frequently  mentioned 
problems  with  the  aids  was  that  they  were  not  user  friendly. 

Even  if  an  aid  has  great  capabilities,  a  bad  soldier  machine 
interface  (SMI)  will  discourage  its  use,  increase  training 
requirements,  increase  errors,  and  in  general  degrade  any 
performance  improvement  it  might  make.  The  SMI  is  so  important 
its  design  should  be  addressed  at  the  beginning  of  development. 

One  suggestion  that  was  made  for  almost  all  of  the  aids  was 
that  input  of  updated  data  should  be  automated.  It  is  very  time 
intensive  to  manually  enter  data.  In  a  time  crunch  it  won't  be 
and  the  systems  won't  be  used. 

Finally,  another  frequently  mentioned  suggestion  is  that  the 
aids  should  be  interconnected.  To  manually  input  the  results 
from  one  aid  into  another  is  time  consuming  and  unnecessary 
effort . 
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APPENDIX  A 

UTILITY  HIERARCHY  AND  QUESTIONS 


A-1 


standardized  Questions 


Note:  The  information  within  brackets  []  indicates  how  the 
evaluator  needs  to  tailor  the  question. 


0.0  Overall  System  Utility 

10.  I  think  an  operational  version  of  the  [System  Name]  is 
good  enough  to  use  in  a  major  training  exercise. 

24.  The  [System  Name]  is  a  valuable  tool  for  [purpose  of 
system] . 

33.  Overall,  the  [System  Name]  is  a  useful  approach  for 
[purpose  of  system] . 

51.  ,yse  of  the  [System  Name]  will  improve  [purpose  of 
system]  performance. 

61.  I  recommend  continued  development  of  the  [System  Name] 
for  operational  use. 

86.  Overall,  the  [System  Name]  improves  [purpose  of  system]. 

1.0  Effect  on  Task  Perfoirmance 
1.1  Process  Quality 

1.1.1  Quality  of  the  Information  (Data  &  Knowledge) 

14.  The  [System  Name]  is  using  the  right  data 
for  [purpose  of  system] . 

36.  I  agree  with  the  [use  the  word  "knowledge” 
or  identify  a  type  of  knowledge  stored  in 
the  knowledge  base]  stored  in  the  [System 
Name]  for  [purpose  of  system] . 

44.  I  agree  with  the  [identify  a  second  type  of 
knowledge  stored  in  the  knowledge  base] 
stored  in  the  [System  Name]  for  [purpose  of 
system] . 

Note:  If  the  above  version  of  question  #44 
is  inappropriate  because  one  can  not  easily 
distinguish  between  the  different  types  of 
knowledge  stored  in  the  knowledge  base,  use 
the  following  version  of  question  #44. 

44.  The  [System  Name]  contains  the  right 
knowledge  for  [purpose  of  system] . 
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65.  I  agree  with  the  [identify  a  third  type  of 
knowledge  stored  in  the  knowledge  base] 
stored  in  the  [System  Name]  for  [purpose  of 
system] . 

Note;  If  the  above  version  of  question  #65 
is  inappropriate  because  one  can  not  easily 
distinguish  between  the  different  types  of 
knowledge  stored  in  the  knowledge  base,  use 
the  following  version  of  question  #65. 

65.  The  [System  Name]  contains  an  adequate 
level  of  expertise  to  support  users 
performing  [purpose  of  system] . 

83.  The  [System  Name]  uses  the  correct 
information  in  producing  its  results. 

^  1.1.2  Quality  of  the  Explanation  Capability/Reasons 

19.  Overall,  the  reasoning  underlying  the  results 
is  acceptable. 

49.  The  [System  Name]  provided  good  reasons  for 
its  results. 

90.  It  is  easy  to  interpret  the  results  of  the 
[System  Name]. 

1.1.3  Quality  of  the  Representation,  Examination,  and 

Modification  of  Knowledge  Stored  in  System 

32.  In  general,  it  is  easy  to  modify  the 
knowledge  stored  in  the  [System  Name]. 

82.  The  [System  Name]  allows  users  to  examine 
the  expert  judgments  on  which  the  system's 
recommendation  is  based. 

1.1.4  Quality  of  the  Problem  Solving  Approach 

11.  The  [System  Name's]  approach  to  representing 
expert  knowledge  for  [purpose  of  system] 
is  acceptable. 

42.  The  [System  Name]  uses  a  logically  sound 
approach  for  [purpose  of  system] . 

58.  The  [System  Name's]  approach  to  [purpose  of 
system]  is  acceptable. 

85.  The  calculations  [or  analysis]  performed  by 
the  [System  Name]  were  helpful. 
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1.1.5  Time  to  Perform  Task(s) 


18.  Using  the  [System  Name]  to  [purpose  of 
system]  was  fast  enough  for  my  needs. 

75.  I  would  feel  comfortable  using  the  [System 
Name]  under  time  pressure. 

81.  Completing  the  task  with  the  [System  Name] 
is  faster  than  current  procedures. 

1.1.6  System  Response  Time 

12.  The  [System  Name]  responds  quickly  to  the 
user's  commands. 

73.  The  [System  Name's]  response  time  is 
acceptable. 

1 . 2  Product  Quality 

1.2.1  Quality  of  the  Results  (i.e.,  products) 

17.  Overall,  the  [System  Name]  provided  me  with 
useful  results. 

34.  I  found  the  [System  Name's]  results 
acceptable. 

59.  The  [System  Name]  supports  the  preparation 
of  high  quality  products. 

76.  The  [System  Name]  would  improve  the  quality 
of  my  work. 

1.2.2  Overall  Confidence 

37.  I  have  alot  of  confidence  in  the  results 
obtained  working  with  the  [System  Name]. 

46.  I  am  confident  that  the  [System  Name]  is 
well-built  technically. 

52.  I  have  alot  of  confidence  in  the  [System 
Name'S]  approach  to  [purpose  of  system]. 

1.2.3  Cost-Effectiveness 

5.  The  [System  Name]  is  cost-effective  because 
the  benefit  of  using  it  is  worth  the  effort. 

80.  The  [System  Name]  provides  users  alot  of 
value  for  their  efforts. 
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2.0  System  Usability 

2.1  General  Ease  of  Use  Questions 

7.  The  displays  are  easy  to  read. 

15.  The  displays  are  easy  to  understand. 

35.  It  was  easy  to  tell  the  [System  Name]  what  to  do. 

55.  The  [System  Name]  is  easy  to  use. 

68.  The  [System  Name's]  input  screens  are  easy  to  use. 

71.  The  mouse  and  keyboard  are  easy  to  use. 

[Note:  The  wording  of  this  question  depends  on  the 
type  of  input  devices  that  the  system  uses.] 

2.2_  Quality  of  the  User's  Mental  Model  of  the  System 

9.  It  was  easy  to  form  a  mental  picture  of  how  the 
[System  Name]  works. 

20.  It  was  easy  to  understand  why  the  results  came 
out  the  way  they  did. 

38.  The  organization  of  menu  items  is  easy  to 
understand . 

56.  It  is  clear  what  to  do  to  get  the  [System  Name]  to 
perform  the  actions  one  wants. 

62.  The  labels  on  the  menu  choices  correctly  describe 
the  choice. 

67.  I  understand  how  to  use  the  [System  Name]  to  do 
[purpose  of  system] . 

95.  The  system  contains  familiar  terms. 

2 . 3  Ease  of  Training 

2.3.1  Learnability 

26.  The  [System  Name]  requires  no  retraining 
for  infrequent  users. 

53.  One  can  learn  to  use  the  [System  Name]  in 
one  two-hour  training  session. 

77.  The  [System  Name]  was  easy  to  learn. 
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2.3.2  On-Line  Help  Function 

3.  The  [System  Name]  has  sufficient  help 
features . 

70.  The  [System  Name's]  help  features  are  easy 
to  use. 

2.3.3  Documentation  -  Not  used  for  AA  Comparator,  but 

the  standard  form  of  the  questions  is  presented 

below. 

-  How  to  use  [System  Name]  is  well  documented. 

-  The  [System  Name's]  User's  Manual  is  easy  to 
understand. 

2.4  Human  Factors  Guidelines  for  Person-Machine  Interaction 

2.4.1  Consistency 

25.  The  [System  Name]  uses  the  same  layout  for 
all  screens. 

48.  The  [System  Name]  presents  similar 

information  at  the  same  place  on  the  screen. 

63 .  The  same  commands  produce  the  same  actions 
throughout  the  [System  Name]. 

2.4.2  Error  Prevention  and  Handling 

16.  The  [System  Name]  helps  to  prevent  errors 
th«  user  might  make  when  using  it. 

40.  The  [System  Name]  provides  immediate  error 
notification. 

64.  The  [System  Name]  is  designed  so  that  it  is 
easy  to  recover  from  errors,  if  they  should 
occur  when  using  it. 

2.4.3  System  Navigation 

41.  It  is''  always  clear  where  the  user  is  in  the 
[System  Name]. 

54.  The  user  can  easily  move  form  one  menu  item 
to  another  without  errors  in  the  [System 
Name] . 

72.  The  user  can  easily  move  to  different  parts 
of  the  [System  Name]  as  required  to  do  the 
tasks . 
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2.4.4  Use  of  Color 


29.  The  [System  Name]  uses  color  in  an 
intuitive  way. 

78.  I  understand  the  meaning  of  the  different 
colors  used  in  the  displays. 

2.4.5  Ease  of  Data  Entry 

30.  I  can  easily  supply  the  information  the 
[System  Name]  asks  me  for. 

50.  It  is  easy  to  enter  data  into  the  [System  Name]. 

2.4.6  Feedback 

23.  The  [System  Name]  provides  feedback  when 
^  it's  processing  user  commands. 

60.  The  [System  Name]  provides  the  user  with 
effective  directions  so  that  one  always 
knows  what  to  do  next. 

2.4.7  Reliability  -  Not  used  for  AA  Comparator,  but 

the  standard  form  of  the  questions  is  presented 

below. 

-  The  number  of  system  failures  is  acceptable. 

-  The  level  of  down  time  is  acceptable. 

-  The  same  inputs  produce  the  same  results. 

2 . 5  Workload 

2.5.1  Strain  (i.e.,  effort) 

6 .  The  user  does  not  have  to  exert  much  mental 
effort  to  use  the  [System  Name]  to  compare 
avenues  of  approach. 

45.  The  [System  Name]  reduces  the  amount  of  work 
required  to  compare  avenues  of  approach. 

74.  The  amount  of  effort  required  to  use  the 
[System  Name]  is  acceptable. 

2.5.2  Memorization 

28.  The  user  does  not  have  to  memorize  commands 
to  use  the  [System  Name]. 

92.  All  necessary  information  is  available  on 
each  screen. 
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2.6  Flexibility  (Discretion  and  User  Control) 

21.  I  felt  in  control  of  the  [System  Name]  when  it 
was  operated. 

69.  The  system  allows  for  adaptation  to  different 
scenarios. 

96.  The  [System  Name]  permits  the  user  to  control 
the  order  in  which  different  activities  are 
done. 

2.7  Functional  Allocation  Between  Person  and  Machine 

22.  The  [System  Name]  supports  those  tasks  requiring 
support  when  [purpose  of  system] . 

88.  The  [System  Name]  is  designed  so  theat  the  right 
activities  are  allocated  to  the  person  and 
machine . 

94.  The  [System  Name]  provides  me  with  the  right 
kind  of  support  for  [purpose  of  system] . 

4 . 0  How  Well  the  System  Fits  In 

4.1  Fit  (i.e.,  Match)  With  User 

4.1.1  Match  With  Users'  Needs 

43.  The  [System  Name]  meets  my  needs  for 
[purpose  of  system] . 

47.  The  [System  Name's]  products  meet  my  needs. 

79.  It  is  easier  to  [purpose  of  system]  using 
the  [System  Name]  than  with  my  current 
procedures . 

4.1.2  Match  With  Users'  Training 

8.  The  [System  Name]  is  designed  to  match  the 
computer  skills  of  Army  personnel  who  would 
use  it. 

84.  The  system's  approach  to  comparing  avenues 
of  approach  matches  how  I  was  trained  to 
perform  this  task. 

4.1.3  Match  With  Users'  Problem-Solving  Approach 

31.  The  [System  Name]  performs  [purpose  of 
system]  the  way  I  do. 
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39.  The  [System  Name's]  approach  to  [purpose  of 
system]  matches  my  idea  of  how  this  task 
should  be  done. 

91.  In  general,  the  [System  Name]  uses  the  same 
information  that  I  use. 

4.2  Fit  (i.e.,  Match)  With  Organization 

4.2.1  Match  With  Doctrine 

13.  The  procedures  used  in  the  [System  Name] 
are  consistent  with  Army  doctrine. 

57.  The  procedures  used  in  [System  Name]  follow 
Army  doctrine. 

4.2.2  Organizational  Fit 

1.  The  [System  Name]  fits  well  in  the 
[organizational  place  for  the  system] . 

87.  From  a  [organizational  place  for  the  system] 
perspective,  the  [System  Name]  is  a  good  fit. 

4.2.3  Effect  on  Information  Flow 

2.  The  [System  Name]  will  facilitate  the  flow 
of  information  in  the  [organizational  place 
for  the  system] . 

89.  The  [System  Name]  will  not  interfere  with 
th&  flow  of  information  in  the 
[organizational  place  for  the  system] . 

4.2.4  Effect  on  Other  People's  Workload 

27.  The  [System  Name]  would  not  increase  the 

amount  of  work  for  other  people  involved  in 
[purpose  of  system] . 

93.  The  [System  Name]  will  decrease  the  workload 
of  other  people  in  the  [organizational  place 

for  the  system] . 

\ 

4.2.5  Attitude  of  Others  (Political  Acceptability) 

4.  Other  people  in  the  [organizational  place  for 
the  system]  will  support  the  [System  Name's] 
implementation . 

66.  My  superiors  would  strongly  favor  the  using 
the  [System  Name] . 
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APPENDIX  B 

EXAMPLE  STUDENT  QUESTIONNAIRE 
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SPECC 

(Space  Enhanced  Command  and  Control  Capabilities) 

SPECC  is  a  (iedsion  aid  that  supports  automated  worldwide  unit  location  functions.  It  provides:  ^ 

•  Worldwide  Automated  Unit  Ucations 
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c  will  Improve  planning  and  execution  performance. 
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i7.  To  Hliat  extent  would  SPECC  support  the  following  tasks? 
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19.  Other  commenu  (e.g.,  limitations,  advantages,  disadvantages,  suggested  additions  and  modlHcatlons). 


APPENDIX  C 

EXAMPLE  DATA  COLLECTOR  QUESTIONNAIRE 


SPECC  '•  n»U  Colltdor  Form 
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Please  circle  (he  appropriate  response. 
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3S.  Thk  scctioa  examints  the  abilitj  of  specific  SPECC  capabilities  to: 

(1)  increajc  situtioaal  awareaeaa, 

(2)  facilitate  a  shared  understanding  of  the  battlefield, 

(3)  assist  in  the  syncfaroniiatioo  of  tactical  operations,  induding  lateral  coordination  and 

(4)  facilitate  piaa  rehearsaL 


For  each  capability  below  please: 

(1)  rate  the  potential  of  this  Hac  of  capability  to  improve  performance  on  each  of  the  4  tasks, 

(2)  rate  how  much  oirrent  SPECC  improves  performance  compared  to  unaided  performance. 

(3)  describe  changes  that  shonJd  be  made  to  SPECC’s  capabilities  to  improve  it 
Put  a  nomher  in  each  blank  Bsiog  the  following  scale. 


No  Slight  Much  Very  Much  Not 

Improvement  Improvement  Improvement  Improvement  Improvement  Applicable 
1  2  3  a  e  > 
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SPECC 

Oiingri  to  SPECC? 
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Maitiraap  display 
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SPECC 

Changes  to  SPECC? 


Woridwide  asKt  tracking 

Potential 

SPECC 

Chaisges  to  SPECC? 


OPORD  Generation/Transmission 

Potential  ____ 

SPECC  _ 

Changes  to  SPECC? _ 
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38.  (Coociaucd) 


For  each  capabilitj  b<3ow  pleiM: 

(1)  rate  th«  potential  oCthiatuiCof  opabilitj  to  improve  performance  oa  each  of  the  4 

(2)  rate  how  much  current  SPECC  Improves  performaocs  compared  to  unaided  performance. 

(3)  describe  changes  that  should  be  made  to  SPECC’s  capabilities  to  improve  iL 
Put  4  number  in  each  bUnk  using  the  following  sak. 


^  Slight  Much  Very  Much 

Improvement  Improvement  Improvement  ‘ImprovemeK  Improvement 
1  2  3  4  5 


Not 

Applicable 

NA 


SPtCC 

CAPAIIUTY 


dcvilop 

srr, 

AWAJIENESS 


Sy?tCHJtONlZI 

TACnCAL 

OPS 


DCV.  SHAIXS 

L7a)QSTAM)INC 

oriATTLmELa 


tzsxjasz 

PLAN 


SfDchroaizatioa  Matrices 


PotendaJ 

SPECC 

Ciangw  to  SPECC? 


Uait/Luf/l^HrsoeBei  Stataa 

Poitatial  _ 

SPSCC  _ 

Chaatea  to  SPECC? 

— 

— 

— 

Joint  Forces  C2  AppUabOtj 

Poteatiai  _ 

SPSCC  _ 

CliaHtw  to  SPECC? 

— 

— 

— 

eXher  C2  featares  (Unit  Stataa,  Task  Org..  Graphics) 

Paicadal  _  ___ 

SPECC  _  _ 

Changes  to  SPECC? 

— 

— 

Woridwide  aatomated  uit  locadons  , 
PotesUiai 

SPECC  _ 

Changes  to  SPECC? 

— 

— 

— 
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3S.  (Coatiaued) 


For  eacli  capability  below  ptease: 

(1)  rate  the  potential  o(  this  tToe  oT capability  to  Improve  performance  oa  eadi  of  tbe  4  rtft’T, 

(2)  rate  bow  mucb  carrent  SPECC  improves  perfonruace  compared  to  uaided  perfonnaoce. 

(3)  describe  changes  that  should  be  made  to  SPECC’s  capabilities  to  improve  it 
Put  a  number  in  each  blank  using  the  following  sale. 


^  Slight  Much  Very  Much  Not 

Improvement  Improvement  Improvement  *  Improvemeiit  Inprovemeat  ApplicabJe 

^  2  3  4  5  NA 


srtcc 

CA/AJUJTY 


DCVSLOP 

srr. 

AWAJtZMSS 


SYNCHXONIZX 

TACTICAL 

on 


MEV.SSAIS) 

UNSCaSTANDlNC 

orsAnumu) 


tZHZAXSX 

PLAN 


Tlmiag  Sybchrooizatioa  (Worldwide  Atomic  Clock) 


Potetfuf 

SPECC 


t*  SPECC? 


Weridwide  Intercnniiectivity  (SaldBtc  Comwfcadons) 


*•  SPECC? 


SPECC 


of  SPECC  Improife  tke  quality  of  your  work  In  Ihc  following 
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